Search for: All records

Research-based assessments (RBAs) allow researchers and practitioners to compare student performance across different contexts and institutions. In recent years, research attention has focused on the student populations these RBAs were initially developed with because much of that research was done with “samples of convenience” that were predominantly white men. Prior research has found that the Force Concept Inventory (FCI) behaved differently for men and women using differential item functioning (DIF) analysis. We extend this research in two ways. First, we test the FCI for DIF across the intersection of gender and race for Asian, Black, Hispanic, White, and White Hispanic men and women. Second, we apply the Eaton and Willoughby five-factor model of the FCI to interpret the results of the DIF analysis. We found large DIF on a large number of FCI items. The patterns of items with large DIF follows the five-factor model. The alignment of DIF with this factor structure, along with the measurement invariance of this factor structure across these ten social identities, indicates that the items on the FCI are likely not biased but are instead measuring real differences in physics knowledge among these groups. We frame these differences as educational debts that society owes to these marginalized groups that physics instruction needs to actively repay. 

Physics instructors and education researchers use research-based assessments (RBAs) to evaluate students' preparation for physics courses. This preparation can cover a wide range of constructs including mathematics and physics content. Using separate mathematics and physics RBAs consumes course time. We are developing a new RBA for introductory mechanics as an online test using both computerized adaptive testing and cognitive diagnostic models. This design allows the adaptive RBA to assess mathematics and physics content knowledge within a single assessment. In this article, we used an evidence-centered design framework to inform the extent to which our models of skills students develop in physics courses fit the data from three mathematics RBAs. Our dataset came from the LASSO platform and includes 3,491 responses from the Calculus Concept Assessment, Calculus Concept Inventory, and Pre-calculus Concept Assessment. Our model included five skills: apply vectors, conceptual relationships, algebra, visualizations, and calculus. The "deterministic inputs, noisy 'and' gate'' (DINA) analyses demonstrated a good fit for the five skills. The classification accuracies for the skills were satisfactory. Including items from the three mathematics RBAs in the item bank for the adaptive RBA will provide a flexible assessment of these skills across mathematics and physics content areas that can adapt to instructors' needs. 

In physics education research, instructors and researchers often use research-based assessments (RBAs) to assess students’ skills and knowledge. In this paper, we support the development of a mechanics cognitive diagnostic to test and implement effective and equitable pedagogies for physics instruction. Adaptive assessments using cognitive diagnostic models provide significant advantages over fixed-length RBAs commonly used in physics education research. As part of a broader project to develop a cognitive diagnostic assessment for introductory mechanics within an evidence-centered design framework, we identified and tested the student models of four skills that cross content areas in introductory physics: apply vectors, conceptual relationships, algebra, and visualizations. We developed the student models in three steps. First, we based the model on learning objectives from instructors. Second, we coded the items on RBAs using the student models. Finally, we then tested and refined this coding using a common cognitive diagnostic model, the deterministic inputs, noisy “and” gate model. The data included 19 889 students who completed either the Force Concept Inventory, Force and Motion Conceptual Evaluation, or Energy and Momentum Conceptual Survey on the LASSO platform. The results indicated a good to adequate fit for the student models with high accuracies for classifying students with many of the skills. The items from these three RBAs do not cover all of the skills in enough detail, however, they will form a useful initial item bank for the development of the mechanics cognitive diagnostic. 

The American Physical Society calls on its members to improve the diversity of physics by supporting an inclusive culture that encourages women and Black, Indigenous, and people of color to become physicists. Introductory physics courses provide opportunities for recruiting and retaining diverse students or enacting policies and cultural practices that disproportionately harm students from minoritized groups. Introductory calculus-based electricity and magnetism courses have received far less attention from researchers than introductory mechanics courses. To better understand the role introductory electricity and magnetism courses play in the lack of diversity in physics, we investigated the intersecting relationships between racism and sexism in inequities in student conceptual knowledge using a quantitative critical framework. The analyses used Bayesian hierarchical linear models to examine students' conceptual knowledge as measured by the Conceptual Survey of Electricity and Magnetism. The data came from the LASSO database and included 3,686 students from 83 calculus-based courses at 16 institutions. The model indicated society owed educational debts in conceptual knowledge due to racism, sexism, or both to Black, Hispanic, Asian, and White Hispanic students and White women. Of these groups, society owed the largest educational debts to Black students. The courses, of which almost all used collaborative instruction (81 of 83) supported by learning assistants (66 of 83), added to the educational debts owed to Black students, maintained the debts owed to Hispanic and White Hispanic students and White women, and mitigated the debts owed to Asian students. 

Research-based assessments (RBAs) measure how well a course achieves discipline-specific outcomes. Educators can use outcomes from RBAs to guide instructional choices and to request resources to implement and sustain instructional transformations. One challenge for using RBAs, however, is a lack of comparative data, particularly given the skew in the research literature toward calculus-based courses at highly selective institutions. In this article, we provide a large-scale dataset and several tools educators in introductory physics courses can use to inform how well their courses foster student conceptual understanding of Newtonian physics. The supplemental materials include this dataset and these tools. Educators and administrators will often target courses with high drop, withdrawal, and failure rates for transformations to student-centered instructional strategies. RBAs and the comparative tools presented herein allow educators to address critiques that the course transformations made the courses “easier” by showing that the transformed course supported physics learning compared to similar courses at other institutions. Educators can also use the tools to track course efficacy over time. 

Education researchers often compare performance across race and gender on research-based assessments of physics knowledge to investigate the impacts of racism and sexism on physics student learning. These investigations' claims rely on research-based assessments providing reliable, unbiased measures of student knowledge across social identity groups. We used classical test theory and differential item functioning (DIF) analysis to examine whether the items on the Force Concept Inventory (FCI) provided unbiased data across social identifiers for race, gender, and their intersections. The data was accessed through the Learning About STEM Student Outcomes platform and included responses from 4,848 students posttests in 152 calculus-based introductory physics courses from 16 institutions. The results indicated that the majority of items (22) on the FCI were biased towards a group. These results point to the need for instrument validation to account for item bias and the identification or development of fair research-based assessments. 

Assessments of students’ attitudes and beliefs often rely on questions with rating scales that ask students the extent to which they agree or disagree with a statement. Unlike traditional physics problems with a single correct answer, rating scale questions often have a spectrum of 5 or more responses, none of which are correct. Researchers have found that responses on rating scale items can generally be treated as continuous and that unless there is good evidence to do otherwise, response categories should not be collapsed [1–3]. We discuss two potential reasons for collapsing response categories (lack of use and redundancy) and how to empirically test for them. To illustrate these methods, we use them on the Colorado Learning Attitudes about Science Survey. We found that students used all the response categories on the CLASS but that three of them were potentially redundant. This led us to conclude that the CLASS should be scored on a 5-point or 3-point scale, rather than the 2-point scale recommended by the instrument developers [4]. More broadly, we recommend the judicious use of data manipulations when scoring assessments and retaining all response categories unless there is a strong rational for collapsing them.